SAMZA-2124: Add Beam API doc to the website by xinyuiscool · Pull Request #948 · apache/samza

xinyuiscool · 2019-03-09T03:19:11Z

Add beam quick start, examples and api docs.

dxichen

Minor comments, thanks for the docs!

dxichen · 2019-03-12T08:35:46Z

docs/learn/documentation/versioned/api/beam-api.md

+
+{% endhighlight %}
+
+To run this Beam program with Samza, you can simply provides "--runner=SamzaRunner" as a program argument. You can follow our [quick start](/startup/quick-start/{{site.version}}/beam.html) to set up your project and run different examples. For more details on writing the Beam program, please refer the comprehensive [Beam programming guide](https://beam.apache.org/documentation/programming-guide/).


s/provides/provide
s/refer the comprehensive/refer to the..

dxichen · 2019-03-12T08:41:48Z

docs/startup/code-examples/versioned/beam.md

+```
+$ deploy/examples/bin/run-beam-standalone.sh org.apache.beam.examples.WordCount \
+    --configFilePath=$PWD/deploy/examples/config/standalone.properties \
+    --inputFile=/Users/xiliu/opensource/samza-beam-examples/pom.xml --output=word-counts.txt \


remove username, I have a patch for these docs here apache/samza-beam-examples#1

I switched to KafkaWordCount, to avoid the batch problems we have.

dxichen · 2019-03-12T08:43:23Z

docs/startup/code-examples/versioned/beam.md

+```
+ $ deploy/examples/bin/run-beam-yarn.sh org.apache.beam.examples.WordCount \
+    --configFilePath=$PWD/deploy/examples/config/yarn.properties \
+    --inputFile=/Users/xiliu/opensource/samza-beam-examples/pom.xml \


remove username

Fixed by switching to kafka.

dxichen · 2019-03-12T08:45:58Z

docs/startup/code-examples/versioned/samza.md

+
+#### Samza SQL API examples
+You can easily create a Samza job declaratively using 
+[Samza SQL](https://samza.apache.org/learn/tutorials/0.14/samza-sql.html).


change version to latest

shanthoosh · 2019-03-12T16:34:01Z

docs/learn/documentation/versioned/api/beam-api.md

+
+### Apache Beam - A Samza’s Perspective
+
+The goal of Samza is to provide large-scale streaming processing capabilities with first-class state support. This does not contradict with Beam. In fact, while Samza lays out a solid foundation for large-scale stateful stream processing, Beam adds the cutting-edge stream processing API and model on top of it. The Beam API and model allows further optimization in the Samza platform, including multi-stage distributed computation and parallel processing on the per-key basis. The performance enhancements from these optimizations will benefit both Samza and its users. Samza can also further improve Beam model by providing various use cases. Adopting Beam provides a solid understanding of the latest data processing technology, and we believe Samza will benefit from it.


s/Adopting Beam provides a solid understanding of the latest data processing technology/ Beam provides cutting-edge data processing capabilities.

shanthoosh · 2019-03-12T16:35:09Z

docs/learn/documentation/versioned/api/beam-api.md

+
+### Introduction
+
+Apache Beam brings an easy-to-use, but powerful API and model for state-of-art stream and batch data processing with portability across a variety of languages. The Beam API and model has the following characteristics:


Minor:
s/but powerful API/ powerful API

Seems better to keep the but. I removed "," to improve readability.

shanthoosh · 2019-03-12T16:35:53Z

docs/learn/documentation/versioned/api/beam-api.md

+
+- *Simple constructs, powerful semantics*: the whole beam API can be simply described by a `Pipeline` object, which captures all your data processing steps from input to output. Beam SDK supports over [20 data IOs](https://beam.apache.org/documentation/io/built-in/), and data transformations from simple [Map](https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/MapElements.html) to complex [Combines and Joins](https://beam.apache.org/releases/javadoc/2.11.0/index.html?org/apache/beam/sdk/transforms/Combine.html).
+
+- *Strong consistency via event-time*: Beam provides advanced [event-time support](https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data) so you can perform windowing and aggregations based on when the events happen, instead of when they are consumed. The event-time mechanism improves the accuracy of processing results, and has repeatability when reprocessing the same data set.


Minor:

s/instead of when they are consumed/instead of arrival time?

s/and has repeatability/and guarantees repeatability in results/

shanthoosh · 2019-03-12T16:38:13Z

docs/startup/code-examples/versioned/beam.md

+
+1. Download and install [Apache Maven](http://maven.apache.org/download.cgi) by following Maven’s [installation guide](http://maven.apache.org/install.html) for your specific operating system.
+
+1. A script named "grid" is included in this project which allows you to easily download and install Zookeeper, Kafka, and Yarn.


I think all the individual line-items in SetUp(Install JDK, install maven, install grid) are numbered with 1. May be it would better to provide them right ordering.

Thanks for the catch. install grid shouldn't be marked a 1. I fixed in the update.

docs/startup/code-examples/versioned/beam.md

dxichen · 2019-03-12T19:19:20Z

LGTM, thanks!

shanthoosh

LGTM.

* SAMZA-2124: Add Beam API doc to the website * Address pr feedback

SAMZA-2124: Add Beam API doc to the website

b116e9f

dxichen reviewed Mar 12, 2019

View reviewed changes

shanthoosh reviewed Mar 12, 2019

View reviewed changes

Address pr feedback

ce462ae

dxichen approved these changes Mar 12, 2019

View reviewed changes

shanthoosh approved these changes Mar 12, 2019

View reviewed changes

shanthoosh merged commit 6711a9f into apache:master Mar 12, 2019

asfgit pushed a commit that referenced this pull request Mar 12, 2019

SAMZA-2124: Add Beam API doc to the website (#948)

3470591

* SAMZA-2124: Add Beam API doc to the website * Address pr feedback

Zhangyx39 pushed a commit to Zhangyx39/samza that referenced this pull request Apr 3, 2019

SAMZA-2124: Add Beam API doc to the website (apache#948)

c5cb62c

* SAMZA-2124: Add Beam API doc to the website * Address pr feedback


		{% endhighlight %}

		To run this Beam program with Samza, you can simply provides "--runner=SamzaRunner" as a program argument. You can follow our [quick start](/startup/quick-start/{{site.version}}/beam.html) to set up your project and run different examples. For more details on writing the Beam program, please refer the comprehensive [Beam programming guide](https://beam.apache.org/documentation/programming-guide/).


		### Apache Beam - A Samza’s Perspective

		The goal of Samza is to provide large-scale streaming processing capabilities with first-class state support. This does not contradict with Beam. In fact, while Samza lays out a solid foundation for large-scale stateful stream processing, Beam adds the cutting-edge stream processing API and model on top of it. The Beam API and model allows further optimization in the Samza platform, including multi-stage distributed computation and parallel processing on the per-key basis. The performance enhancements from these optimizations will benefit both Samza and its users. Samza can also further improve Beam model by providing various use cases. Adopting Beam provides a solid understanding of the latest data processing technology, and we believe Samza will benefit from it. No newline at end of file


		### Introduction

		Apache Beam brings an easy-to-use, but powerful API and model for state-of-art stream and batch data processing with portability across a variety of languages. The Beam API and model has the following characteristics:


		- Simple constructs, powerful semantics: the whole beam API can be simply described by a `Pipeline` object, which captures all your data processing steps from input to output. Beam SDK supports over [20 data IOs](https://beam.apache.org/documentation/io/built-in/), and data transformations from simple [Map](https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/MapElements.html) to complex [Combines and Joins](https://beam.apache.org/releases/javadoc/2.11.0/index.html?org/apache/beam/sdk/transforms/Combine.html).

		- Strong consistency via event-time: Beam provides advanced [event-time support](https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data) so you can perform windowing and aggregations based on when the events happen, instead of when they are consumed. The event-time mechanism improves the accuracy of processing results, and has repeatability when reprocessing the same data set.


		1. Download and install [Apache Maven](http://maven.apache.org/download.cgi) by following Maven’s [installation guide](http://maven.apache.org/install.html) for your specific operating system.

		1. A script named "grid" is included in this project which allows you to easily download and install Zookeeper, Kafka, and Yarn.

Conversation

xinyuiscool commented Mar 9, 2019

Uh oh!

dxichen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dxichen commented Mar 12, 2019

Uh oh!

shanthoosh left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants